Enhancement of compressed images

ABSTRACT

A method is provided for recovering an image defined as a function of image coordinates of an image space from compressed data that is a function of transform coordinates in a transform space comprising: a) transforming the compressed data using at least one partial transform to generate a set of intermediate coefficients in a space intermediate between the transform space and said image space; b) adjusting the value of at least one intermediate coefficient and using said adjusted value to generate a set of adjusted intermediate coefficients; and c) recovering the image by transforming the set of adjusted intermediate coefficients to the image space with at least one additional partial transform.

RELATED APPLICATIONS

The present application is a US national application of PCT/IL99/00025,filed on Jan. 14, 1999 and claims the benefit under 35 U.S.C. Section119(e) of U.S. Provisional Application No. 60/115,863 filed on Jan. 13,1999.

FIELD OF THE INVENTION

The present invention relates to processing compressed data and inparticular to processing compressed data representing images.

BACKGROUND OF THE INVENTION

Still and moving images generally contain large amounts of information.For example, an RGB color image comprising 2000×2000 pixels requires 12megabytes of data if the intensity of each RGB color component of eachpixel is coded with 8 bits. Transmission and storage of this amount ofdata even for single images is generally impractical for most purposes.For example, if images of this size are to be transmitted at typicalmoving picture rates of thirty frames per second, then 360 megabytes ofimage data must be transmitted every second. This is clearly a verylarge rate of data transmission. Therefore, data representing an imageis generally “compressed”. Compression significantly reduces the amountof data representing the image in comparison to the amount of datarequired to represent the image by coding RGB intensities (orintensities of a different set of color components used for definingcolor) for each pixel of an image. Hereinafter, the intensity of a colorcomponent used to define color of a pixel in an image is referred to asa “pixel value”.

Different techniques exist for compressing data and in particular forcompressing data representing an image. In many of these techniques theimage is first partitioned into a plurality of contiguous,non-overlapping, sub-images or tiles wherein each sub-image extends overa portion of the area of the full image. Variation of image intensityand color over the area of a sub-image is generally limited. For eachcolor component used to define color in the image, pixel values of eachsub-image are transformed into a set of values, hereinafter referred toas “transform coefficients”, in a transform space. Generally, the pixelvalues are transformed by operating on the pixel values with a unitaryseparable transform. The pixel values are recoverable from the transformcoefficients by operating on the transform coefficients with an inverseof the transform. Among well known separable unitary transforms used inprocessing image data are the Fourier, cosine, sine and Hadamardtransforms.

Generally, the number of transform coefficients needed to effectivelyrecover the pixel values is considerably less than the number of pixelvalues. The amount of data needed to code the transform coefficients istherefore generally significantly less than the amount of data needed tocode the pixel values. Representing the image using the transformcoefficients therefore requires considerably less data than representingthe image by coding color component intensities for each pixel. Thetransform thus succeeds in compressing the image data.

Usually, the transform coefficients are quantized by dividing eachtransform coefficient by a “quantizer”. Any remainder resulting from thedivision is rounded up or down and the transform coefficient is replacedby the quotient. Different transform coefficients are generallyquantized with quantizers of different magnitudes. Often the magnitudeof a quantizer is determined as a function of an error margin for thevalue of the transform coefficient that the quantizer is used toquantize, which error margin is estimated from known error margins ofthe pixel values.

Transform space data representing the image after quantizing thetransform coefficients comprises the quantized transform coefficientsand the quantizer, or a way of determining the quantizer, for eachquantized transform coefficient. When recovering the image, i.e. whenrecovering the pixel values, a transform coefficient, hereinafterreferred to as a “recovered transform coefficient”, is recovered fromeach quantized transform coefficient by multiplying the quantizedtransform coefficient by its quantizer. The recovered transformcoefficients are operated on by an inverse of the transform used togenerate the transform coefficients, to determine the pixel values.

Quantization increases the compression of the image data beyond thatachieved with the transform. Quantization reduces the range of differentnumbers representing the transform coefficients, which reduces thenumber of bits required to code the coefficients. In addition some ofthe coefficients (those that are less than one half their quantizer) arequantized to zero, which reduces the number of coefficients used torecover the original pixel values. Increasing quantizer magnitudesdecreases the possible different values of quantized transformcoefficients and increases the number of transform coefficients that arequantized to zero. Increasing quantizer magnitudes therefore generallyincreases the extent to which image data is compressed.

However, unlike compression resulting from a non-quantized unitarytransform, compression from quantization is “lossy”. Informationcontained in the remainders discarded in the quantization process islost and the larger the quantizer magnitudes the more lossy is thecompression. The recovered transform coefficients are different from thetransform coefficients by a quantization error. As a result, pixelvalues generated using quantized transform coefficients differ from theoriginal pixel values.

The steps in the industry standard JPEG system for compressing color andgray tone images illustrates a typical data compression procedure for acolor image. In JPEG a color image is generally partitioned into squaretiles of 8×8 pixels. Color of the image is usually defined using YUVcolor components, so that each pixel has Y, U and V pixel values. Foreach tile, for each of the YUV color components, the pixel values of thecolor component are transformed into a set of transform coefficientsusing a discrete cosine transform. The transform coefficients are thenquantized. Usually the transform coefficients associated with higherspatial frequencies are quantized with larger quantizers.

To illustrate the process, let Y(x,y) represent luminance pixel valuesin a tile, where x and y are integer coordinates that locate a pixel inthe tile as being in the x-th row and y-th column of the tile. Thetransform coefficients are functions of two integer transform spacecoordinates, “u” and “v” that are conjugate coordinates of x and yrespectively.

If C(u,v) represents the transform coefficients, then${C\left( {u,v} \right)} = {\sum\limits_{y = 0}^{7}{\sum\limits_{x = 0}^{7}{{{DCT}\left( {u,v,x,y} \right)}{L\left( {x,y} \right)}}}}$where DCT(u,v,x,y) symbolically represents the discrete cosinetransform. Writing the discrete cosine transform explicitly gives:${{C\left( {u,v} \right)} = {{A(u)}{A(v)}{\sum\limits_{y = 0}^{7}{\sum\limits_{x = 0}^{7}{{L\left( {x,y} \right)}\cos\;\left( {\left( {{2x} + 1} \right)u\;{\pi/16}} \right)\cos\;\left( {\left( {{2y} + 1} \right)v\;{\pi/16}} \right)}}}}},\;{{{where}\mspace{14mu}{A(w)}} = {1/\sqrt{2}}}$for w=0 and 1 otherwise. Each C(u,v) is then quantized with a quantizer,Q(u,v), yielding a quantized transform coefficientQC(u,v)=INT{[C(u,v)±Q(u,v)/2]/Q(u,v)}, where the + is used if C(u,v)>0,the minus is used if C(u,v)<0 and INT represents rounding the result tothe nearest integer.

To recover pixel values for luminance from the QC(u,v), recoveredtransform coefficients, RC(u,v), are calculated whereRC(u,v)=QC(u,v)Q(u,v). The recovered transform coefficients are thentransformed back with the inverse discrete cosine transform representedby IDCT(x,y,u,v), so that recovered luminance values RY(x,y) may bewritten:${{RY}\left( {x,y} \right)} = {\sum\limits_{u = 0}^{7}{\sum\limits_{v = 0}^{7}{{{IDCT}\left( {x,y,u,v} \right)}{{{RC}\left( {u,v} \right)}.}}}}$Explicitly,${{RY}\left( {x,y} \right)} = {{A(u)}{A(v)}{\sum\limits_{u = 0}^{7}{\sum\limits_{v = 0}^{7}{{{RC}\left( {u,v} \right)}\cos\;\left( {\left( {{2x} + 1} \right)u\;{\pi/16}} \right)\cos\;{\left( {\left( {{2y} + 1} \right)v\;{\pi/16}} \right).}}}}}$The recovered value for luminance RY(x,y) is generally not equal toY(x,y) because the quantization process has resulted in a loss ofinformation so that generally RC(u,v)≠C(u,v).

In many instances, information lost in quantizing transform coefficientsdoes not affect the quality of an image recovered from the transformcoefficients to an extent that renders the image unusable for thepurpose for which it was intended. However, in many instances the lossof information results in objectionable degradation of the recoveredimage. Often, unwanted artifacts are generated in the recovered imageand often the image is degraded unacceptably. For example, a set ofcompressed transform coefficients might be useable to provide a good“thumbnail” image of a scene but as a result of information loss thetransform coefficients may be totally inadequate to provide an enlargedimage of the scene.

Techniques exist for adjusting images recovered from compressed data soas to reduce effects on the quality of the recovered image that resultfrom information loss and error in the compressed data. Some techniquesaddress specific types of defects or artifacts, for example “blockingartifacts”, that are generated in images recovered from compressed datathat has lost information. Other techniques apply algorithms to reducepixel to pixel discontinuities or to improve image sharpness by edgeenhancement. Most such techniques have limitations. For example,edge-enhancing algorithms typically increase noise in a recovered imageand smoothing algorithms tend to blur an image. There is a need for newtechniques for recovering images from compressed data that compensatefor information loss or errors in the compressed data.

Whereas the above discussion has focused on data representing twodimensional images comprising pixels and associated pixel values ofcolor components or gray tones, it should be realized that thediscussion also applies to three dimensional data sets comprising voxelsand corresponding “voxel values”. Quite generally, the presentdiscussion is germane to n-dimensional images. An n-dimensional image isdefined as a set of values, hereinafter referred to as “image values”,that are dependent on n independent coordinates, which coordinatesdefine an n-dimensional image space. Furthermore it should be noted thatthe definition of an n-dimensional image is quite general and ann-dimensional image is not restricted to a video image, it means, amongother things, a sound image as well. For example a three dimensionalsound image might be the notes in a sound file that are a function oftime and two stereo channels.

Compression of data representing an n-dimensional image can be similarto compression of a two dimensional video image comprising pixel valuesas illustrated with the JPEG example. Image values of the n-dimensionalimage are transformed preferably by a separable unitary transform into aset of transform coefficients dependent upon transform space coordinatesof an n-dimensional transform space. The transform coefficients are thenquantized using appropriate quantizers. Each of the transform spacecoordinates is conjugate to a different one of the coordinates of theimage space. An image is recovered from transform coefficients bydequantizing the quantized transform coefficients to determine recoveredtransform coefficients and operating on the recovered transformcoefficients with an inverse of the unitary transform. As in the case oftwo dimensional images the recovered image is degraded by loss ofinformation, or error in the recovered transform coefficients andtechniques are needed to compensate for the information loss.

SUMMARY OF THE INVENTION

An aspect of preferred embodiments of the present invention relates toproviding a method for adjusting compressed data comprising quantizedtransform coefficients generated by transforming image data, preferably,with a separable unitary transform. The method is useable to moderatedeleterious effects on an image recovered from the compressed data thatresult from information loss and/or error in the compressed data.

Aspects of some preferred embodiments of the present invention relate toprocessing compressed data that represents an n-dimensional image.

In accordance with aspects of some preferred embodiments of the presentinvention the method relates to processing compressed data generatedfrom two and three-dimensional images.

Aspects of some preferred embodiments of the present invention relate toprocessing compressed data generated by compressing image data using theJPEG system and derivatives of the JPEG system such as MPEG and relatedand similar compression schemes such as H261, H263, H323, PX64 and HDTV.

Preferably, the transform used to compress the data is separable so thatit can be decomposed into a set of partial transforms that operatesequentially to transform the image values into the transformcoefficients. A first partial transform operates on the image data andgenerates a set of first partial transform coefficients. A secondpartial unitary transform operates on the set of first partial transformcoefficients to generate a set of second partial transform coefficients.The procedure continues with each partial transform operating on partialtransform coefficients generated by the preceding partial transform. Theresults of the last partial transform are the transform coefficients. Inthe process of transforming the data each partial transform converts atleast one coordinate of the image space into its conjugate transformcoordinate in the transform space. The transform can thus be decomposedinto, at most, n partial transforms in which each partial transformconverts one different image space coordinate into its transform spacecoordinate.

The inverse transform can preferably be decomposed into correspondingpartial inverse transforms in which each partial inverse transform is aninverse of a corresponding partial transform. In preferred embodimentsof the present invention, the partial inverse transforms operatesequentially to “roll back” the results of the partial transforms, eachpartial inverse transform rolling back the results of a correspondingpartial transform. If a partial transform generates a set of thirdpartial transform coefficients from a set of second partial transformcoefficients then its corresponding partial inverse transform generatesthe set of second transform coefficients from the set of third transformcoefficients.

Each partial transform coefficient, hereinafter referred to as a“partial coefficient”, is a function of at least one transform spacecoordinate and at least one image space coordinate. Partial coefficientsgenerated by a same sequence of partial transforms and/or partialinverse transforms are functions of the same transform space coordinatesand the same image space coordinates and are said to belong to a sameset of partial coefficients. While in preferred embodiments of thepresent invention both transforms have the above characteristics, inother preferred embodiments of the present invention only the inversetransform is required to be separable in that the total inversetransform can be achieved by operation on the compressed data by asequence of partial inverse transforms.

In accordance with a preferred embodiment of the present invention thecompressed data is adjusted by adjusting at least one partialcoefficient to provide at least one adjusted partial coefficient. The atleast one adjusted partial coefficient is used in place of the at leastone partial coefficient in recovering an image from the transformcoefficients.

For at least one partial coefficient an adjustment limit is preferablydetermined that is used to set an upper and lower bound for the amountby which the partial coefficient may be increased or decreased by anadjustment. The range of values determined by the upper and lower boundare hereinafter referred to as an “adjustment range”. The adjustmentcomprises using an adjustment algorithm to assign a value to the atleast one partial coefficient that lies in the adjustment range. Usingthe algorithm does not necessarily result in a change in the value ofthe at least one partial coefficient. In some cases the result of usingthe algorithm is to leave the value of the at least one partialcoefficient unchanged.

According to an adjustment algorithm used in some preferred embodimentsof the present invention, the value of the at least one partialcoefficient is adjusted so as to moderate at least one apparent anomalyin values of partial coefficients belonging to the set of partialcoefficients to which the partial coefficient belongs.

According to an adjustment algorithm used in some preferred embodimentsof the present invention, the value of the at least one partialcoefficient is adjusted so as to conform to ratios between other partialcoefficients in the same set of partial coefficients.

According to an adjustment algorithm used in some preferred embodimentsof the present invention, the value of the at least one partialcoefficient is adjusted to conform to a trend in changes in values ofpartial coefficients as a function of at least one of the transformspace coordinates or one of the image space coordinates.

According to an adjustment algorithm used in some preferred embodimentsof the present invention, the value of the at least one partialcoefficient is adjusted to conform to a predetermined pattern relatingvalues of partial coefficients.

For compressed data representing color images, such as, for example,data compressed using JPEG, MPEG and derivative compression systems, insome preferred embodiments of the present invention, the value of atleast one partial coefficient for one color component is adjustedresponsive to at least one value of a transform coefficient or a partialcoefficient for a different color component. For example a partialcoefficient for a chrominance component U or V may be adjusted usingpartial coefficients for luminance Y.

Quite generally, if first and second sets of compressed data are fromrelated images then recovered coefficients and/or partial coefficientsgenerated from the first set of compressed data may be adjusted, inaccordance with a preferred embodiment of the present invention,responsive to recovered coefficients and or partial coefficients of thesecond set of compressed data.

In some preferred embodiments of the present invention, if the value ofat least one partial coefficient is changed in a set of partialcoefficients, the at least one changed value is checked for consistencywith the value of at least one recovered transform coefficient. In somepreferred embodiments of the present invention, the check is performedby transforming the “adjusted” set of partial coefficients using thepartial transform that transforms the set of partial coefficients intothe set of transform coefficients (i.e. the inverse of the partialtransform that transforms the set of transform coefficients into the setof partial coefficients). At least one value of the transformed set ofadjusted partial coefficients is then compared with at least onetransform coefficient using an appropriate metric to determine if theset of adjusted partial coefficients are consistent with the transformcoefficients.

In some preferred embodiments of the present invention, if the value ofat least one partial coefficient is changed in a first set of partialcoefficients, the at least one changed value is checked for consistencywith the value of at least one partial coefficient in a second set ofpartial coefficients. In some preferred embodiments of the presentinvention, the check is performed by transforming the adjusted first setof partial coefficients using the partial transform or partial inversetransform that transforms the first set of partial coefficients into thesecond set of partial coefficients. The transformed set of adjustedfirst partial coefficients is then compared with the second set ofpartial coefficients using an appropriate metric to determine if thefirst and second sets of partial coefficients are consistent.

In some preferred embodiments of the present invention an adjustmentrange for a partial coefficient is determined responsive to values ofother partial coefficients in the same set of partial coefficients.

In some preferred embodiments of the present invention an adjustmentrange for a partial coefficient is determined responsive to themagnitude of the quantizers used to quantize transform coefficients.

In some preferred embodiments of the present invention the value of atleast one recovered transform coefficient is adjusted using anadjustment algorithm prior to operating on the transform coefficientswith a partial transform. In some preferred embodiments of the presentinvention the at least one recovered transform coefficient is adjustedaccording to an algorithm responsive to the value of the transformcoefficient's quantizer. In some preferred embodiments of the presentinvention, the at least one recovered transform coefficient is adjustedaccording to an algorithm responsive to the value of a recoveredtransform coefficient or the value of partial coefficient.

According to an adjustment algorithm used in some preferred embodimentsof the present invention, the value of the at least one transformcoefficient is adjusted to conform to a predetermined pattern relatingvalues of transform coefficients.

For compressed data representing color images, in some preferredembodiments of the present invention, the value of at least onetransform coefficient for one color component is adjusted responsive toat least one value of a transform coefficient for a different colorcomponent.

Generally, as in the case of partial coefficients, recoveredcoefficients generated from a first set of compressed data may beadjusted, in accordance with a preferred embodiment of the presentinvention, responsive to recovered coefficients of a second set ofcompressed data.

Using an algorithm to adjust the value of at least one transformcoefficient does not necessarily change the value of the at least onetransform coefficient. In some cases the result of using the algorithmis to leave the value of the at least one transform coefficientunchanged.

The inventors have found that for JPEG compressed images, or imagescompressed using derivative and similar compression techniques,adjusting partial coefficients in accordance with a preferred embodimentof the present invention, reduces noise in the recovered image to agreater extent than it decreases contributions to the recovered imagefrom high spatial frequency transform coefficients. Therefore, thequality of the recovered image is enhanced as a result of noisereduction but with less loss of sharpness than usually results fromnoise reduction techniques used in image processing. Furthermore, inaccordance with a preferred embodiment of the present invention, partialcoefficients are adjusted as described above. An image recovered usingthe adjusted partial coefficients is then enhanced with a sharpnessenhancing procedure, in accordance with some embodiments of theinvention. The resulting recovered image, in accordance with a preferredembodiment of the present invention, is enhanced by both noise reductionand edge enhancement.

There is therefore provided in accordance with a preferred embodiment ofthe present invention a method for recovering an image defined as afunction of image coordinates of an image space from compressed datathat is a function of transform coordinates in a transform spacecomprising: a) transforming the compressed data using at least onepartial transform to generate a set of intermediate coefficients in aspace intermediate between the transform space and the image space; b)adjusting the value of at least one intermediate coefficient and usingthe adjusted value to generate a set of adjusted intermediatecoefficients; and c) recovering the image by transforming the set ofadjusted intermediate coefficients to the image space with at least oneadditional partial transform.

Preferably, adjusting the value of at least one partial coefficientcomprises: a) determining an adjustment range; b) determining a value inthe adjustment range; and c) setting the value of the intermediatecoefficient equal to the determined value.

Preferably, the compressed data comprises information for determining aset of quantized values and a quantizer for each quantized value asfunctions of coordinates of the transform space, and transforming thedata comprises determining a first set of coefficients in the transformspace using the quantizers and the quantized values and transforming thefirst set of coefficients with the at least one partial transform.

Determining the first set of coefficients preferably comprisesdetermining the product of each quantized value with its quantizer anddetermining the first set of coefficients as a set comprising all theproducts. Alternatively, determining the first set of coefficientspreferably comprises determining the product of each quantized valuewith its quantizer, adjusting the value of at least one of the productsand determining the first set of coefficients as a set comprising allthe products.

In some preferred embodiments of the present invention, determining anadjustment range comprises determining at least one adjustment limit anddetermining the adjustment range responsive to the adjustment limit.

Preferably, determining at least one adjustment limit comprisestransforming the set of quantizers with the at least one transform togenerate a set of transformed quantizers in the intermediate space anddetermining an adjustment limit for at least one intermediatecoefficient responsive to at least one transformed quantizer of the setof transformed quantizers.

Preferably, determining at least one adjustment limit comprisesadjusting the value of at least one quantizer responsive to values ofthe first coefficients prior to transforming the set of quantizers.Adjusting the value of at least one quantizer preferably comprisesdetermining whether all of the first set of coefficients are zero for avalue of a single coordinate of the transform space greater than acertain value and constant values for all other transform spacecoordinates and if so, setting the values of the quantizerscorresponding to the zero coefficients equal to a same value.Preferably, the same value is equal to the maximum of the coefficientsfor all values of the single coordinate and the constant values for allother coordinates.

Additionally or alternatively, the at least one adjustment limit for theat least one intermediate coefficient is preferably determinedresponsive to the transformed quantizer having the same coordinates inthe intermediate space as the at least one intermediate coefficient.Preferably, the adjustment limit is equal to the correspondingtransformed quantizer multiplied by a fraction less than one.Preferably, the fraction is equal to 0.5.

In some preferred embodiments of the present invention, determining atleast one adjustment limit comprises determining an adjustment limit forat least one intermediate coefficient responsive to the value of atleast one of the intermediate coefficients.

Preferably, determining the at least one adjustment limit comprisesdetermining the difference between a maximum and minimum intermediatecoefficient in a plurality of intermediate coefficients in the set ofintermediate coefficients and determining the adjustment limitresponsive to the difference. Preferably, the plurality of intermediatecoefficients comprises intermediate coefficients in a neighborhood ofthe at least one intermediate coefficient.

Alternatively and preferably, all but one of the coordinates of any twoof the plurality of intermediate coefficients are the same and theplurality of intermediate coefficients includes the at least oneintermediate coefficient.

Alternatively or additionally the adjustment limit is preferably equalto the difference multiplied by a fraction less than one. Preferably,the fraction is equal to 0.5.

Some preferred embodiments of the present invention comprise testing theset of adjusted intermediate coefficients for consistency with the firstset of coefficients in the transform space.

Preferably, testing the set of adjusted intermediate coefficientscomprises transforming the set of adjusted intermediate coefficientsinto a second set of coefficients in the transform space and comparingcoefficients of the first set with coefficients of the second set todetermine if the set of adjusted coefficients is consistent with thefirst set of coefficients. Preferably, comparing coefficients comprisesdetermining how close the value of at least one coefficient of the firstset is to the value of at least one coefficient of the second set.Determining how close the value of at least one coefficient of the firstset is to the value of at least one coefficient of the second setpreferably comprises using a metric.

In some preferred embodiments of the present invention, if the set ofintermediate coefficients is not consistent with the first set ofcoefficients, the value at least one adjusted intermediate coefficientof the set of adjusted intermediate coefficients is readjusted togenerate a readjusted set of intermediate coefficients so that the setof readjusted intermediate coefficients is consistent with the first setof coefficients.

In some preferred embodiments of the present invention adjusting the atleast one intermediate coefficient comprises adjusting the at least oneintermediate coefficient responsive to coefficients in a neighborhood ofthe at least one intermediate coefficient. Preferably, the neighborhoodcomprises a plurality of intermediate coefficients, including the atleast one intermediate coefficient, for which any two intermediatecoefficients have same values for all the same coordinates except one.

In some preferred embodiments of the present invention, adjusting the atleast one intermediate coefficient comprises adjusting the at least oneintermediate coefficient responsive to a ratio between a partialcoefficient and a function of other partial coefficients.

In some preferred embodiments of the present invention, adjusting the atleast one intermediate coefficient comprises adjusting the at least onepartial coefficient responsive to a trend in changes of values ofpartial coefficients as a function of changes in at least one coordinateof the intermediate space.

In some preferred embodiments of the present invention, adjusting the atleast one intermediate coefficient comprises adjusting the at least onepartial coefficient responsive to an expected trend in changes of valuesof partial coefficients as a function of changes in at least onecoordinate of the intermediate space.

In some preferred embodiments of the present invention, adjusting the atleast one intermediate coefficient comprises adjusting the at least onepartial coefficient responsive to a predetermined template. Preferably,the template identifies an edge in the recovered image and comprisingsmoothing the recovered image on either side of the identified edge.

In some preferred embodiments of the present invention, adjusting thevalue of the at least one intermediate coefficient comprises adjustingthe value of the at least one intermediate coefficient responsive todata from different compressed data that is related to the compresseddata. Preferably, the image in the image space comprises values for afirst color component of a color image in the image space and thedifferent compressed data comprises data for generating a second colorcomponent image of the color image. Additionally or alternatively, thedifferent compressed data identify an edge in the recovered image. Somepreferred embodiments of the present invention in which an edge isidentified comprise smoothing the image on either side of the edge.

Some preferred embodiments of the present invention comprise testing theset of adjusted intermediate coefficients for consistency with a firstset of intermediate coefficients defined in an intermediate spacedifferent from the intermediate space in which the adjusted intermediatecoefficients are defined.

Preferably, testing the set of adjusted intermediate coefficientscomprises transforming the set of adjusted intermediate coefficientsinto a second set of intermediate coefficients in the differentintermediate space and comparing intermediate coefficients of the firstset with intermediate coefficients of the second set to determine if theset of adjusted coefficients is consistent with the first set ofintermediate coefficients.

Preferably, comparing coefficients comprises determining how close thevalue of at least one intermediate coefficient of the first set is tothe value of at least one intermediate coefficient of the second set.Determining how close the value of at least one intermediate coefficientof the first set is to the value of at least one intermediatecoefficient of the second set preferably comprises using a metric.

If the set of adjusted intermediate coefficients is not consistent withthe first set of intermediate coefficients, in some preferredembodiments of the present invention, the value at least one adjustedintermediate coefficient of the set of adjusted intermediatecoefficients is readjusted to generate a readjusted set of intermediatecoefficients so that the set of readjusted intermediate coefficients isconsistent with the first set of intermediate coefficients.

There is further provided, in accordance with a preferred embodiment ofthe present invention, a method for recovering an image defined as afunction of image coordinates in an image space from compressed datathat is a function of transform coordinates in a transform space themethod comprising: a) using the data to generate a first set ofrecovered transform coefficients in the transform space; b) determiningan adjustment range for at least one recovered transform coefficient; c)replacing the at least one recovered transform coefficient by a value inthe adjustment range responsive to recovered transform coefficients in aneighborhood comprising less than all of the recovered transformcoefficients of the at least one recovered transform coefficient, togenerate an adjusted set of recovered transform coefficients; and d)recovering the image by transforming the adjusted set of recoveredtransform coefficients to the image space.

Preferably, the neighborhood comprises a plurality of recoveredtransform coefficients including the at least one recovered coefficient,for which any two recovered coefficients have same values for all thesame coordinates except one.

There is further provided, in accordance with a preferred embodiment ofthe present invention, a method for recovering an image defined in animage space from compressed data that comprises quantized values and aquantizer for each quantized value that are functions of transformcoordinates in a transform space the method comprising: a) determining aset of recovered transform coefficients in which each recoveredtransform coefficient is a product of a quantized value with itsquantizer; b) determining an adjustment range having an upper and lowerlimit for at least one recovered transform coefficient of the first setof recovered coefficients, wherein the difference between the upper andlower limit is less than the magnitude of the quantizer used todetermine the at least one recovered coefficient; c) replacing the atleast one recovered transform coefficient by a value in the adjustmentrange to generate an adjusted set of recovered transform coefficients;and d) recovering the image by transforming the adjusted set ofrecovered transform coefficients to the image space.

There is further provided, in accordance with a preferred embodiment ofthe present invention, a method for recovering an image defined in animage space from a first set of compressed data that comprises quantizedvalues and a quantizer for each quantized value that are functions oftransform coordinates in a transform space the method comprising: a)determining a first set of recovered transform coefficients in whicheach recovered transform coefficient is a product of a quantized valuewith its quantizer; b) determining an adjustment range for at least onerecovered coefficient; c) replacing the at least one transformcoefficient with a value in the adjustment range responsive to recoveredtransform coefficients generated using a second set of relatedcompressed data to generate an adjusted set of recovered transformcoefficients; and d) recovering the image by transforming the adjustedset of recovered transform coefficients to the image space. Preferably,the first and second sets of compressed data are generated fromdifferent color component images of a color image.

There is further provided, in accordance with a preferred embodiment ofthe present invention, a method for adjusting an image having aplurality of image values defined in an image space, the image valuesbeing generated from compressed data comprising quantized values and aquantizer for each quantized value that are functions of transformcoordinates in a transform space the method comprising: determining anadjustment range for at least one image value of the plurality of imagevalues responsive to the quantizers; determining a value in theadjustment range; adjusting the image by replacing the at least oneimage value with the determined value.

Preferably, determining an adjustment range comprises determining anaverage of a plurality of the quantizers. Alternatively, determining anadjustment range preferably comprises determining a maximum quantizer ina plurality of the quantizers. Alternatively, determining an adjustmentrange preferably comprises determining a minimum quantizer in aplurality of the quantizers.

In some preferred embodiments of the present invention, the plurality ofthe quantizers comprises all quantizers in the compressed data.

In some preferred embodiments of the present invention, determining avalue comprises determining a value responsive to image values of adifferent related image. Preferably, the image and the different relatedimage are color component images of a multicolor color image.

Some preferred embodiments of the present invention, comprise smoothingthe recovered image.

In some preferred embodiments of the present invention, the image is animage defined in a three dimensional image space.

In some preferred embodiments of the present invention, the image is animage defined in a two dimensional space.

The compressed data in some preferred embodiments of the presentinvention is data generated using a unitary separable transform totransform an image in the image space into a set of values in thetransform space.

In some preferred embodiments of the present invention, the compresseddata is generated using a JPEG compression method.

In some preferred embodiments of the present invention, the compresseddata is generated using an MPEG compression method.

In some preferred embodiments of the present invention, the compresseddata is generated using a PX64 compression method.

In some preferred embodiments of the present invention, the compresseddata is generated using an H261 compression method.

In some preferred embodiments of the present invention, the compresseddata is generated using an H263 compression method.

In some preferred embodiments of the present invention, the compresseddata is generated using an H323 compression method.

In some preferred embodiments of the present invention, the compresseddata is generated using an HDTV compression method.

The invention will be more clearly understood by reference to thefollowing description of preferred embodiments thereof read inconjunction with the figures attached hereto.

BRIEF DESCRIPTION OF FIGURES

FIGS. 1A–1B show a flow diagram schematically illustrating a procedurefor recovering an image from two dimensional data compressed data, suchas JPEG and MPEG compressed data, compressed using a two dimensionaldiscrete cosine transform, in accordance with a preferred embodiment ofthe present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIGS. 1A and 1B show a flow diagram of a procedure 20 for recovering animage from two dimensional compressed data 22 (shown in FIG. 1A). Data22 represents compressed data that is compressed from a two dimensionalimage defined in an image space using any of many different suitabletransforms and quantization procedures. However, for clarity andefficacy of exposition it is assumed that data 22 represents data for acolor image compressed using a JPEG compression procedure.

Assume, for example, that compressed data 22 represents luminance valuesof pixels in an 8 pixel by 8 pixel tile, of a plurality of pixel tilesinto which a color image is partitioned. Compressed data 22 thereforecomprises information sufficient to determine values for sixty fourquantized transform coefficients QC(u,v) and a quantizer Q(u,v) for eachQC(u,v). It should be noted that Q(u,v) is not generally a constant andis often a function of u and v.

Each quantized transform QC(u,v) is generated from a correspondingtransform coefficient C(u,v) by quantizing transform coefficient C(u,v)using quantizer Q(u,v). The C(u,v) are generated in JPEG by transformingthe luminance values of the pixels in the tile with the discrete cosinetransform. If I(x,y) represents the luminance of the pixel in the x-throw and y-th column of the tile then${{C\left( {u,v} \right)} = {\sum\limits_{x = 0}^{7}{\sum\limits_{y = 0}^{7}{{{DCT}\left( {u,v,x,y} \right)}{I\left( {x,y} \right)}}}}},$where DCT(u,v,x,y) represent the two dimensional discrete cosinetransform. However since DCT(u,v,x,y) is separable, the equation forC(u,v) can be written${{C\left( {u,v} \right)} = {\sum\limits_{x = 0}^{7}{\sum\limits_{y = 0}^{7}{{{DCT}_{1}\left( {u,x} \right)}{{DCT}_{2}\left( {v,y} \right)}{I\left( {x,y} \right)}}}}},$where DCT₁(u,x) and DCT₂(v,y) are partial transforms that convert imagespace coordinates, x and y, hereinafter referred to as spatialcoordinates, into transform space coordinates u and v respectively. Theinverse discrete cosine transform IDCT(x,y,u,v), can similarly bewritten as IDCT₂(y,v)IDCT₁(x,u).

Procedure 20 begins processing compressed data 22 in a step 24 in whichcompressed data 22 is used to generate a data set 26 of recoveredtransform coefficients RC(u,v). It is often convenient to visualize dataset 26 as a matrix of elements whose values are the magnitudes of theRC(u,v), where u is the row and v is the column of the matrix in whichthe value of RC(u,v) is located. Following common convention the elementin the upper left corner of the matrix is the magnitude of RC(0,0). Datasets comprising the QC(u,v), Q(u,v) and other related data sets aresimilarly conveniently visualized. An RC(u,v) matrix 28 is schematicallyshown in inset 30 of FIG. 2.

In step 32 a decision is made whether or not to adjust the recoveredtransform coefficients RC(u,v). In some preferred embodiments of thepresent invention the decision to adjust or not to adjust is maderesponsive to values of RC(u,v) and/or values of Q(u,v). For example inaccordance with a preferred embodiment of the present invention onlyvalues of RC(u,v) that are equal to ±1 are adjusted, since relativeerrors due to quantization in these recovered coefficients are generallygreater than relative errors in higher valued recovered coefficients.

If procedure 20 does adjust recovered transform coefficients RC(u,v),procedure 20 advances to step 34 in which the value of at least onerecovered transform coefficient RC(u,v) is adjusted using at least oneadjustment algorithm.

According to one adjustment algorithm, in accordance with a preferredembodiment of the present invention, the value of RC(u,v) is set equalto [QC(u,v)Q(u,v)−Q(u,v)(u+v)/32] instead of QC(u,v)Q(u,v). A recoveredtransform coefficient RC(u,v) can differ from the transform coefficientC(u,v) from which it was quantized by an amount that can vary up to amaximum amount, hereinafter referred to as a “quantization error”, thatis one half the magnitude of the quantizer used to quantize C(u,v).Therefore, the effect of this algorithm is to bias the value of arecovered transform coefficient RC(u,v) to lower values of a range ofpossible values for the transform coefficient C(u,v) from which QC(u,v)was generated by an amount equal to a fraction of its quantizationerror. Furthermore the bias is larger for higher spatial frequencyrecovered transform coefficients than for lower spatial frequencyrecovered transform coefficients (i.e. for C(u,v) with larger values foru+v). In a variation of this algorithm, RC(u,v) is set equal to[QC(u,v)Q(u,v)−αQ(u,v)], where α is a fraction less than one. Accordingto this algorithm for adjusting recovered transform coefficients, eachrecovered transform coefficient is reduced by the same fraction of itsquantization error. Preferably α is greater than 0.5. More preferably, αis equal to 0.75.

Generally, the magnitudes of cosine transform coefficients of an imagedecrease substantially monotonically with increasing spatial frequency,especially for relatively small homogeneous images. If recoveredtransform coefficients RC(u,v) do not exhibit this expected behaviorthey are adjusted to reduce deviation from the expected behavior usinganother adjustment algorithm in accordance with a preferred embodimentof the present invention. In accordance with a preferred embodiment ofthe present invention, RC(u,v) are increased or decreased by amounts upto their quantization errors, so that to an extent possible, for aconstant value for one of the arguments u or v, the absolute values ofRC(u,v), or alternatively the values of RC(u,v), monotonically decreasewith increase in the other argument. In some preferred embodiments ofthe present invention RC(u,v) that are equal to zero are not included inthe adjustment procedure.

For example, a straight line can be fitted by regression to the RC(u,v)in a row. If the regression line has a negative slope, individualRC(x,v) are adjusted within their quantization errors so that they areas close as possible to the regression line. If the regression line doesnot have a negative slope, values of RC(x,v) are not adjusted.Alternatively, for example, if the regression line does not have anegative slope, a regression line with a negative slope can be searchedfor using a subset of the RC(x,v). If a suitable regression line isfound then values of the RC(x,v) in the subset are adjusted to exhibitmonotonically decreasing values. For example, a regression curve can befitted only for those RC(x,v) having spatial frequencies greater thanthe spatial frequency of the maximum RC(x,v) in the row. Other ways ofdefining a negative slope for adjusting the R(x,v) will occur to personsof the art.

The decrease in magnitude of transform coefficients with increasingspatial frequency is a characteristic common to many transforms. Thetransform coefficients can be considered projections of an image onbasis functions of the transform. The basis functions exhibit differentrates of spatial variation and can be arranged in an ordered set inwhich each basis function in the set exhibits a higher rate, i.e.frequency, of spatial variation than the basis function that precedes itin the ordered set. The first basis function, often called the DCfunction, is a constant and its corresponding transform coefficient isan average of the image over a spatial range in which the image isdefined. At some basis function spatial frequency, the frequency ofvariation of a basis function is generally larger than a dominant rateof change in the image. From that spatial frequency on, the magnitudesof the transform coefficients are generally monotonically decreasingwith increasing spatial frequency. For relatively homogeneous images(the tiles in a JPEG image are generally small enough so that they aresubstantially homogeneous) the transform coefficients are substantiallymonotonically decreasing from the DC transform coefficient on.

The application of an adjustment algorithm to data set 26 of recoveredtransform coefficients RC(u,v) generates a data set 36 of adjustedrecovered transform coefficients RC*(u,v). Procedure 20 then advances toa step 38 in which data set 36 is transformed by one of the partialinverse discrete cosine transforms, IDCT₁(x,u) or IDCT₂(y,v), into adata set 40 of partial coefficients PRC(x,v) or PRC(u,y). For clarityand economy of presentation it is assumed that in step 38, data set 36comprising RC*(u,v) is transformed using IDCT₁(x,u) so that data set 40comprises partial coefficients PRC(x,v).

If the decision in decision step 32 is not to adjust at least onerecovered transform coefficient RC(u,v) then procedure 20 advancesdirectly from decision step 32 to step 38 and partial transformIDCT₁(x,u) transforms data set 26 into data set 40 instead oftransforming data set 36 into data set 40.

Transforming data set 40 of partial coefficients PRC(x,v) with partialtransform DCT₁ (u,x) regenerates the RC(u,v), or the RC*(u,v), fromwhich the PRC(x,v) are generated.

A visualization of PRC(x,v) data set 40 as a matrix 42 is schematicallyshown in inset 44 of FIG. 1. The spatial coordinate x indicates the rowand the transform coordinate v indicates the column of matrix 42 inwhich an element PRC(x,v) is located. PRC(0,0) is in the upper leftcorner of matrix 42. If recovered transform coefficients RC(u,v) wereequal to transform coefficients C(u,v), i.e. if there is no error orinformation loss in compressed data 26, and no RC(u,v) is adjusted, thenthe PRC(x,v) in a row “X” would be the cosine transform of I(x,y) forx=X along the y-axis. Increasing column number v corresponds toincreasing spatial frequency along the y-axis of I(x,y).

Following generation of data set 40 of partial coefficients PRC(x,v), instep 46, an adjustment range is preferably determined for at least onePRC(x,v). An adjustment range for a PRC(x,v) can be determined indifferent ways, in accordance with various preferred embodiments of thepresent invention.

Assume PRC(x,v) has an adjustment limit, ALIM(x,v). An adjustmentalgorithm, in accordance with a preferred embodiment of the presentinvention, adjusts PRC(x,v) by assigning PRC(x,v) a value in anadjustment range of values defined by PRC(x,v)±ALIM(x,v). Whereas thenoted defined range of values is symmetric, it is possible, and may beadvantageous, in accordance with a preferred embodiment of the presentinvention, to define a range that is asymmetric about PRC(x,v).Furthermore, it should be noted that adjustment limits and adjustmentranges can be computed in different ways and have different values fordifferent adjustment algorithms.

In accordance with a preferred embodiment of the present inventionadjustment limits for PRC(x,v) are determined by transforming thequantizers Q(u,v) of compressed data 22 with IDCT₁(x,u) to generate aset of partial quantizers PQ(x,v). ALIM(x,v) is then preferably setequal to α(x,v)PQ(x,v), where α(x,v) is preferably a fraction less thanor equal to one half. More preferably α is set equal to one half.

In some preferred embodiments of the present invention the Q(u,v) areadjusted to provide a set of adjusted quantizers Q*(u,v) that are usedinstead of Q(u,v) to calculate the ALIM(x,v). The Q*(u,v) are not usedto calculate recovered coefficients RC(u,v) only the values foradjustment ranges.

In some preferred embodiments of the present invention, adjustments toQ(u,v) are made responsive to values of quantized transform coefficientsQC(u,v) in compressed data 22. For example, assume that quantizedtransform coefficients QC(u,v) are equal to zero for v>V in a row u=U.Assume that the maximum value for the QC(u,v) in row U occurs in columnW, so that the maximum quantized transform coefficient in row U isQC(U,W). Then, in accordance with a preferred embodiment of the presentinvention, the quantizers Q(u,v) for u=U and v>V are set equal toQC(U,W) to generate an adjusted set of quantizers Q*(u,v) with which todetermine adjustment limits ALIM(x,v). Since it is expected that QC(u,v)decrease monotonically with v, it is reasonable to limit an adjustmentlimit for the QC(U,v) that are zero to the value of the largest non-zeroQC(U,v).

Adjustment limits ALIM(x,v) can also be determined responsive to valuesof partial coefficients PRC(x,v), in accordance with a preferredembodiment of the present invention. For example, an adjustment limitALIM(x,v) for x=X and v=V can be determined to be equal to a fractionless than one of the difference between a maximum and a minimum partialcoefficient PRC(x,v) in row X or column V or a surround of “location”X,V. Preferably the fraction is greater than or equal to 0.5. Morepreferably the fraction is equal to 0.5. Other methods for determiningadjustment limits ALIM(x,v) will occur to persons of the art.

In yet other preferred embodiments of the present invention ALIM(x,v) isdetermined to be equal to a product of a fraction less than one timesthe minimum PRC(x,v) in either row x or column v.

Following determination of an adjustment limit ALIM(x,v) and anadjustment range in step 46 for at least one PRC(x,v), the at least onePRC(x,v) is then adjusted, in a step 48, using at least one appropriateadjustment algorithm. The adjustment algorithm is used to determine anew value for each of the at least one PRC(x,v), which new value lieswithin the adjustment range of the PRC(x,v) determined from theadjustment limit ALIM(x,v). The value of the PRC(x,v) is then set equalto the new value. The result of adjusting the at least one PRC(x,v) isgeneration of an adjusted data set 50 of partial transform coefficientsPRC*(x,v).

Whereas the at least one adjustment algorithm is described as usedsubject to the constraint of an adjustment limit, it is possible, inaccordance with a preferred embodiment of the present invention, to usealgorithms that are not constrained by an adjustment limit.

It should be noted that a new value for a PRC(x,v) determined by analgorithm can be equal to an “old” value of the PRC(x,v) and thattherefore in some circumstances, in accordance with a preferredembodiment of the present invention, some or all values in data set 50of adjusted transform coefficients PRC*(x,v) are identical tocorresponding values of data set 40 of transform coefficients PRC(x,v).

According to one adjustment algorithm, in accordance with a preferredembodiment of the present invention, PRC(x,v) in at least one row (i.e.constant x) are increased or decreased by amounts limited by at leastone adjustment limit so that to an extent possible the values PRC(x,v)decrease monotonically with increasing column number (i.e. increasingv). This can be achieved, for example, using a regression linedetermination and fitting technique similar to the one described abovefor adjusting recovered transform coefficients RC(u,v) so that thevalues of the recovered transform coefficients decrease substantiallymonotonically with increasing column or row number.

According to another adjustment algorithm, in accordance with apreferred embodiment of the present invention, PRC(x,v) in at least onerow are adjusted to moderate an anomalous value of a PRC(x,v) in therow. In some preferred embodiments of the present invention the value ofthe PRC(x,v) is tested for anomaly by determining an average of valuesof PRC(x,v) in the row and comparing the value of the PRC(x,v) with theaverage. For example, if AV represents the determined average andALIM(x,v) represents a symmetric adjustment limit for the PRC(x,v) (i.e.PRC(x,v) is adjustable in a range PRC(x,v)±ALIM(x,v)), then if|(PRC(x,v)−AV)|≧ALIM(x,v) the value of the PRC(x,v) is consideredanomalous. In some preferred embodiments of the present invention when aPRC(x,v) is anomalous, a low pass filter is applied to the PRC(x,v) andadjacent PRC(x,v) in the row in order to moderate the anomaly. In somepreferred embodiments of the present invention, the value of ananomalous PRC(x,v) is multiplied by an appropriate factor less than oneto reduce the anomaly. In some preferred embodiments of the presentinvention an average AV is calculated according to${AV} = {\left( {{1/2}M} \right)*{\sum\limits_{m = 1}^{M}{\left\lbrack {{{PRC}\left( {x,{v - m}} \right)} + {{PRC}\left( {x,{v + m}} \right)}} \right\rbrack.}}}$

In some preferred embodiments of the present invention anomalies thatare moderated are anomalies that are identified in a value of a firstPRC(x,v) by comparing the value of the first PRC(x,v) with values ofother PRC(x,v) that are in the same column as the first PRC(x,v). Insome preferred embodiments of the present invention the value of a firstPRC(x,v) is compared to values of other PRC(x,v) in an area of thematrix surrounding the first PRC(x,v) to determine if the value of thefirst PRC(x,v) is anomalous.

According to another adjustment algorithm, in accordance with apreferred embodiment of the present invention, the value of a particularPRC(x,v) is adjusted responsive to a ratio between other PRC(x,v). Forexample, generally values of PRC(x,v) for column numbers greater than 2are less reliable than values of PRC(x,v) for column numbers less thanor equal to two. This occurs, generally, because higher spatialfrequency transform coefficients C(u,v) are generally smaller than lowerfrequency transform coefficients C(u,v) and because the higher frequencytransform coefficients are usually quantized with larger quantizers thanlower spatial frequency transform coefficients C(u,v). Accordingly, insome preferred embodiments of the present invention, PRC(x,v) in highernumber columns are adjusted in response to a ratio between a highercolumn number PRC(x,v) and a value that is determined form lower columnnumber PRC(x,v).

For example, assume that PRC(x,v) for a column v=V are to be adjustedand that V≧3. According to a preferred embodiment of the presentinvention a row X is determined for which the value PRC(x,V) is amaximum. A value “VAL(X)” is set equal to the average of, PRC(X,0) andPRC(X,1) in row X, and an adjustment ratio AR=PRC(X,V)VAL(X) iscalculated. The value of at least one other PRC(x,V) in column V is thenadjusted by replacing it with a new value “NPRC(x,V)” that satisfies therelation[PRC(x,V)−ALIM(x,V)]≦NPRC(x,V)≦[PRC(x,V)+ALIM(x,V)] andminimizes [NPRC(x,V)−VAL(x)AR]. In these relations VAL(x) is the averageof PRC(x,0) and PRC(x,1) and ALIM(x,V) is the adjustment limit ofPRC(x,V).

According to another adjustment algorithm, in accordance with apreferred embodiment of the present invention, PRC(x,v) in a same row Xare set to zero if the value PRC(X,v) satisfies the relation:ALIM(X,v)≦PRC(X,v)≦+ALIM(X,v) for all v. In some preferred embodimentsof the present invention, if the above relation is satisfied by all butone or two of the PRC(X,v) then a low pass filter is applied to PRC(X,v)to moderate variations in PRC(X,v).

According to other adjustment algorithms, in accordance with preferredembodiments of the present invention, a partial coefficient or atransform coefficient for one color component image of a color image isadjusted in response to at least one transform coefficient or partialcoefficient of a different color component image of the color image.

For example in JPEG, chrominance components U and V are often coded forevery other pixel of a color image while luminance Y is coded for everypixel, i.e. Y is sampled at twice the rate as U and V in the colorimage. Typically, for an 8 by 8 tile of pixels in the color image, Ywill have 64 values while U and V will each have only sixteen values. Asa result, luminance Y will be specified by sixty-four recoveredtransform coefficients for the tile while U and V will each be specifiedby sixteen recovered transform coefficients for the same tile. Each rowor column of recovered luminance transform coefficients Y compriseseight coefficients while each row or column of recovered chrominancetransform coefficients, U or V, comprises only four coefficients.Recovered Y transform coefficients for the tile will therefore comprisesixty four recovered coefficients and recovered U and V transformcoefficients will comprise 16 recovered coefficients. Using anadjustment algorithm, in accordance with a preferred embodiment of thepresent invention, the four recovered transform coefficients for achrominance component are extrapolated to eight transform coefficients,responsive to values of the chrominance coefficients and values of theluminance coefficients. Extrapolating the four chrominance coefficientsto eight chrominance coefficients provides an estimate of chrominancecoefficients that might result from a finer sampling of the chrominancecomponent of the image.

In accordance with some preferred embodiments of the present invention,recovered U coefficients (or, of course, recovered V coefficients orboth recovered U and V coefficients) are extrapolated to eight recoveredU coefficients that have values that reflect a trend in the values ofthe Y coefficients. For example, assume that a row of Y recoveredcoefficients for a tile of a color image have values 100, 60, 30, 30,20, 10, 0, 0 and that corresponding U recovered coefficients for thetile have values 60, 40, 10, 0. Then an extrapolated set of eightrecovered U coefficients, in accordance with a preferred embodiment ofthe present invention might have values 70, 50, 40, 40, 15, 5, 0, 0.

In accordance with some preferred embodiments of the present inventionan edge that is well defined by data for one color component image of acolor image is used to locate an edge in recovered coefficients and/orpartial coefficients or in a recovered image of another color componentimage of the color image. Once the edged is located, in some preferredembodiments of the present invention the recovered coefficients and/orthe partial coefficients and/or the recovered image are smoothed oneither side of the located edge. This preserves the edge and therebysharpness of the recovered image while reducing noise in the recoveredimage.

Whereas the above example is given for recovered transform coefficients,the example applies equally well to adjusting a partial transformcoefficient for one color component using recovered partial coefficientsor transform coefficients for a different color component.

Furthermore, adjusting recovered coefficients and/or partialcoefficients for one set of compressed data responsive to recoveredcoefficients and/or partial coefficients in a related set of compresseddata is not limited to color component images of color images, inaccordance with a preferred embodiment of the present invention. Quitegenerally, recovered coefficients and or partial coefficients generatedfrom a first set of compressed data may be adjusted, in accordance witha preferred embodiment of the present invention, responsive to recoveredcoefficients and/or partial coefficients of a related second set ofcompressed data.

According to other adjustment algorithms, in accordance with preferredembodiments of the present invention, the value of at least one partialcoefficient is adjusted to conform to a predetermined pattern,hereinafter referred to as a “template”, that defines relationshipsbetween a set of partial coefficients.

A template comprises a set of values and a variance for each value,hereinafter referred to respectively as a “template” value and a“template” variance. Each template value and its template variancedefines a range of values for a different partial coefficient of a setof partial coefficients. The template values are compared, in accordancewith a preferred embodiment of the present invention, to the set ofpartial coefficients using a metric responsive to template variance andadjustment limit of the partial coefficients. The comparison is used todetermine to what degree relationships between the template values andrelationships between the partial coefficients are similar. If,according to the metric, the template and the set of partialcoefficients are considered to match, i.e. relationships between thetemplate values and relationships between the partial coefficients aresufficiently similar, then the partial coefficients are adjusted toimprove the match.

For example, a template for row x=X of JPEG partial coefficients i.e.PRC(X,v), might comprise eight template values and variances that arefunctions of column number v. For the row, let the template values berepresented by T(X,v) and the template variances be represented byTALIM(X,v). Then, in accordance with a preferred embodiment of thepresent invention, a suitable “matching” metric to determine the degreeto which the row of partial coefficients RC(X,v) and the template valuesT(X,v) match, is${``{MATCH}"} = {\sum\limits_{v}{\left\lbrack {{{PRC}\left( {X,v} \right)} - {a\;{T\left( {X,v} \right)}}} \right\rbrack^{2}/{\left\lbrack {{{TALIM}\left( {X,v} \right)}^{2} + {{ALIM}\left( {X,v} \right)}^{2}} \right\rbrack.}}}$In the equation, α is a suitable scaling constant that best removes anyconstant factor between values of PRC(X,v) and T(X,v). If MATCH is lessthan an appropriate threshold, then the PRC(X,v) and the T(X,v) areconsidered a match. If there is a match then, in accordance with apreferred embodiment of the present invention, at least one value of thePRC(X,v) is adjusted within limits determined by its adjustment limitR(X,v) to minimize MATCH.

In accordance with some preferred embodiments of the present invention,a template matching algorithm is used to test a set of partialcoefficients for a match with each of a plurality of different templatesto determine which, if any, of the templates best matches the set ofcoefficients. In some preferred embodiments of the present invention thebest matching template is chosen for use in adjusting the partialcoefficients. For example, in the JPEG example, MATCH might becalculated for each of a plurality of different templates and thetemplate that provides the smallest value for MATCH will be the templateused to adjust at least one PRC(X,v).

In other preferred embodiments of the present invention, if more thanone template is found to match a set of partial coefficients, asdetermined by a suitable matching metric, all of the matching templatesare used to adjust at least one of the partial coefficients. Preferably,this is performed by forming a weighted average of the matchingtemplates and using values of the weighted average to adjust at leastone of the partial coefficients. For example, assume that “m” templatesT_(i)(X,v) are found to match the JPEG partial coefficients PRC(X,v). Ifa weighted average of the matching templates is represented by WT(X,v)then${{WT}\left( {X,v} \right)} = {\sum\limits_{i = 0}^{m}{\beta_{i}{T_{i}\left( {X,v} \right)}}}$where β₁ are appropriate weighting factors and each PRC(X,v) ispreferably adjusted responsive to WT(X,v).

Preferably, in accordance with a preferred embodiment of the presentinvention, some of the templates that are tested for a match with a setof recovered transform coefficients are templates that correspond topossible edges in the image from which the compressed data is generated.When a match is found between a set of partial coefficients and atemplate that represents an edge, the match results in identifying theposition of an edge in an image recovered using the partialcoefficients. In accordance with a preferred embodiment of the presentinvention, pixels in the recovered image on either side of theidentified edge are then adjusted using a low pass filter that does notoperate simultaneously on pixels on both sides of the edge. Thisprocedure reduces noise in the recovered image without reducingsharpness in the image.

Whereas the above example is given for partial coefficients, the exampleapplies equally well to adjusting recovered transform coefficients andadjusting recovered pixel values.

Variations of the adjustment algorithms discussed above and otheralgorithms for adjusting partial transform coefficients will occur topersons of the art.

In a preferred embodiment of the present invention, following generationof adjusted data set 50, data set 50 is preferably transformed “back”,in a step 52 with partial transform DCT₁ (u,x) to generate a data set 54of regenerated transform coefficients RC**(u,v). Values of RC**(u,v) arechecked for consistency with values of RC*(u,v) or RC (u,v) in a step 56by comparing values of RC**(u,v) with values of RC*(u,v) or RC (u,v)using an appropriate metric. Preferably, the metric is responsive todifferences between values of RC**(u,v) and values of RC(u,v) or RC(u,v)and relevant quantization errors and/or adjustment ranges. For example,in some preferred embodiments of the present invention, consistencyrequires that 2|RC**(u,v)−RC(u,v)|/Q(u,v)<1 (Q(u,v) is the quantizer forRC(u,v)).

In decision step 58, if the check for consistency is negative, thenprocedure 20 returns to step 48 and a new value for at least onePRC*(x,v) of data set 50 is determined using appropriate adjustmentalgorithms. For example, in accordance with a preferred embodiment ofthe present invention a new value for a PRC*(x,v) is determined bysubtracting from PRC*(x,v) an amount equal to α[PRC*(x,v)−PRC(x,v)]where α is a fraction less than one. Data set 50 is then againpreferably checked for consistency with RC*(u,v) or RC (u,v).Preferably, new values for at least one PRC*(x,v) of data set 50 aredetermined until data set 50 satisfies the consistency check performedin step 56.

If the check for consistency is positive, decision step 58 preferablyadvances procedure 20 to a step 60 (shown in FIG. 2B). In step 60RC**(u,v) data set 54 is transformed to a data set 62 of partialcoefficients PRC(u,y) using IDCT₂(y,v). Data set 62 is then preferablyadjusted and checked in a sequence of steps represented by a processstep 64 to produce a data set 66 of adjusted partial coefficientsPRC*(u,y). Process step 64 represents a series of step similar to theseries of steps used to adjust and check data set 40 comprising PRC(x,v)to produce data set 50 comprising PRC*(x,v). However data set PRC(u,y)can be checked for consistency, in accordance with a preferredembodiment of the present invention, not only with RC(u,v) and/orRC*(u,v) but also with data set 50 of adjusted partial coefficientsPRC*(x,v).

In step 68 data set 66 is transformed with partial inverse transformIDCT₁(x,u) to generate recovered image 70 comprising pixel valuesRI(x,y).

The inventors have found that processing compressed data 22 inaccordance with a preferred embodiment of the present inventionselectively suppresses noise in recovered image 70 while affectingsharpness of recovered image 70 only moderately. As a result recoveredimage 70 can be edge enhanced by adjusting pixel values RI(x,y) so as toprovide an image that has improved sharpness and reduced noise.Therefore, preferably, procedure 20 comprises a step 72 in whichrecovered image 70 is edge enhanced to provide a sharpened recoveredimage 74 comprising pixel values RI*(x,y).

In adjusting pixel values RI(x,y) of image 70 to generate enhanced image74, preferably an adjustment limit is defined for each pixel valueRI(x,y) that is adjusted, which adjustment limit is used to define anadjustment range for the pixel. Each pixel values RI*(x,y) of enhancedimage 74 is preferably constrained to have a value that lies within theadjustment range determined for its corresponding pixel value RI(x,y) inrecovered image 70. Preferably adjustment limits for pixel valuesRI*(x,y) are defined responsive to Q(u,v). In some preferred embodimentsof the present invention an adjustment limit for RI(x,y) is determinedto be equal to an average of values of all or a subset of Q(u,v)multiplied by a fraction less than one. In some preferred embodiments ofthe present invention an adjustment limit is set equal to a maximum or aminimum value of all or a subset of Q(u,v) multiplied by a fraction lessthan one. Preferably the fraction in either case is less than one half.

It should be noted that, in accordance with a preferred embodiment ofthe present invention, it is not required to generate any particularcombination of data sets to generate a recovered image from compresseddata. For example, in generating recovered image 70 from compressed data22 it is possible, in accordance with a preferred embodiment of thepresent invention, to generate and adjust only data sets 26 and 36.Recovered image 70 is then generated directly from data set 36 withoutgenerating and adjusting partial coefficients. It is also possible, inaccordance with a preferred embodiment of the present invention, togenerate recovered image 70 directly from data set 50 without generatingdata sets 62 or 66.

Quite generally, in accordance with preferred embodiments of the presentinvention, different combinations of transforms and inverse transformsand permutations of a combination of transforms and inverse transformscan be used to process compressed data. Furthermore, in processingcompressed data in accordance with a preferred embodiment of the presentinvention, any data set generated from the compressed data can beadjusted responsive to a consistency check with any other data setgenerated from the compressed data (including a set of recoveredtransform coefficients generated from the data set).

For example, in accordance with preferred embodiments of the presentinvention different combinations of inverse transforms IDCT₁ and IDCT₂and transforms DCT₁ and DCT₂ can be used in processing compressed data22 It is possible to apply the transform IDCT₁(x,u)·DCT₂(v,y) to dataset 66 comprising PRC*(u,y) to check the consistency of PRC*(u,y) withadjusted partial coefficients PRC*(x,v) of data set 50. It is alsopossible to transform recovered image 70 of pixel values RI(x,y), orimage 74 comprising pixel values RI*(x,y) with DCT₁(u,x)·DCT₂(v,y) andcheck the resulting data set for consistency with data set 26 comprisingthe “original” recovered transform coefficients RC(u,v).

It should also be noted that different adjustment algorithms can beapplied, in accordance with a preferred embodiment of the present, todifferent data sets generated from the compressed data and that morethan one adjustment algorithm can be applied to the same data set.

Whereas the invention has been described for a two dimensional imagecompressed using JPEG, the invention applies similarly to n-dimensionalimages in general and to various different types of n-dimensional imagesincluding non-video images such as sound files. The invention alsoapplies to n-dimensional images that are compressed using a separableunitary transform other than a cosine transform.

A preferred embodiment of the present invention is described in aprovisional application entitled “Enhancement of Compressed Images” andappendix thereof submitted to the US Patent Office on Jan. 13, 1999, inthe name of the same inventors of the present invention, the disclosureof which is incorporated herein by reference.

In the claims and description of the present application, each of theverbs, “comprise”, “include” and has, and conjugates thereof, are usedto indicate that the object or objects of the verb are not necessarily acomplete listing of all the components, elements or parts of the subjector subjects of the verb.

The present invention has been described using non-limiting detaileddescriptions of preferred embodiments thereof that are provided by wayof example and are not intended to limit the scope of the invention.Variations of embodiments described will occur to persons of the art.The scope of the invention is limited only by the following claims:

1. A method for recovering an image defined as a function of imagecoordinates of an image space from compressed data that is a function oftransform coordinates in a transform space comprising: a) transformingsaid compressed data using at least one partial transform thattransforms at least one but not all of the transform coordinates into acorresponding image coordinate to generate a set of intermediatecoefficients which are functions of at least one coordinate from theimage space and at least one coordinate from the transform space in aspace intermediate between said transform space and said image spacesaid intermediate space being defined by at least one coordinate fromthe image space and at least one coordinate from the transform space; b)adjusting the value of at least one intermediate coefficient and usingsaid adjusted value to generate a set of adjusted intermediatecoefficients; and c) recovering said image by transforming said set ofadjusted intermediate coefficients to said image space with at least oneadditional partial transform.
 2. A method according to claim 1 whereinadjusting the value of at least one partial coefficient comprises: a)determining an adjustment range; b) determining a value in saidadjustment range; and c) setting the value of said intermediatecoefficient equal to said determined value.
 3. A method according toclaim 2 wherein said compressed data comprises information fordetermining a set of quantized values and a quantizer for each quantizedvalue as functions of coordinates of said transform space, and whereintransforming said data comprises determining a first set of coefficientsin said transform space using said quantizers and said quantized valuesand transforming said first set of coefficients with said at least onepartial transform.
 4. A method according to claim 3 wherein determiningsaid first set of coefficients comprises determining the product of eachquantized value with its quantizer and determining said first set ofcoefficients as a set comprising all said products.
 5. A methodaccording to claim 3 wherein determining said first set of coefficientscomprises determining the product of each quantized value with itsquantizer, adjusting the value of at least one of said products anddetermining said first set of coefficients as a set comprising all saidproducts.
 6. A method according to claim 3 wherein determining anadjustment range comprises determining at least one adjustment limit anddetermining said adjustment range responsive to said adjustment limit.7. A method according to claim 6 wherein determining at least oneadjustment limit comprises transforming said set of quantizers with saidat least one transform to generate a set of transformed quantizers insaid intermediate space and determining an adjustment limit for at leastone intermediate coefficient responsive to at least one transformedquantizer of said set of transformed quantizers.
 8. A method accordingto claim 7 comprising adjusting the value of at least one quantizerresponsive to values of said first coefficients prior to transformingsaid set of quantizers.
 9. A method according to claim 8 whereinadjusting the value of at least one quantizer comprises determiningwhether all of said first set of coefficients are zero for a value of asingle coordinate of said transform space greater than a certain valueand constant values for all other transform space coordinates and if so,setting the values of the quantizers corresponding to said zerocoefficients equal to a same value.
 10. A method according to claim 9wherein said same value is equal to the maximum of said coefficients forall values of said single coordinate and said constant values for allother coordinates.
 11. A method according to claim 7 wherein said atleast one adjustment limit for said at least one intermediatecoefficient is determined responsive to the transformed quantizer havingthe same coordinates in the intermediate space as the at least oneintermediate coefficient.
 12. A method according to claim 11 whereinsaid adjustment limit is equal to said corresponding transformedquantizer multiplied by a fraction less than one.
 13. A method accordingto claim 12 wherein said fraction is equal to 0.5.
 14. A methodaccording to claim 6 wherein determining at least one adjustment limitcomprises determining an adjustment limit for at least one intermediatecoefficient responsive to the value of at least one of said intermediatecoefficients.
 15. A method according to claim 14 wherein determiningsaid at least one adjustment limit comprises determining the differencebetween a maximum and minimum intermediate coefficient in a plurality ofintermediate coefficients in said set of intermediate coefficients anddetermining said adjustment limit responsive to said difference.
 16. Amethod according to claim 15 wherein said plurality of intermediatecoefficients comprises intermediate coefficients in a neighborhood ofsaid at least one intermediate coefficient.
 17. A method according toclaim 15 wherein all but one of the coordinates of any two of theplurality of intermediate coefficients are the same and wherein theplurality of intermediate coefficients includes the at least oneintermediate coefficient.
 18. A method according to claim 15 whereinsaid adjustment limit is equal to said difference multiplied by afraction less than one.
 19. A method according to claim 16 wherein saidfraction is equal to 0.5.
 20. A method according to claim 3 comprisingtesting said set of adjusted intermediate coefficients for consistencywith said first set of coefficients in said transform space.
 21. Amethod according to claim 20 wherein testing said set of adjustedintermediate coefficients comprises transforming said set of adjustedintermediate coefficients into a second set of coefficients in saidtransform space and comparing coefficients of said first set withcoefficients of said second set to determine if said set of adjustedcoefficients is consistent with said first set of coefficients.
 22. Amethod according to claim 21 wherein comparing coefficients comprisesdetermining how close the value of at least one coefficient of saidfirst set is to the value of at least one coefficient of said secondset.
 23. A method according to claim 22 wherein determining how closethe value of at least one coefficient of said first set is to the valueof at least one coefficient of said second set comprises using a metric.24. A method according to claim 20 wherein, if said set of intermediatecoefficients is not consistent with said first set of coefficients, thevalue at least one adjusted intermediate coefficient of said set ofadjusted intermediate coefficients is readjusted to generate areadjusted set of intermediate coefficients so that said set ofreadjusted intermediate coefficients is consistent with said first setof coefficients.
 25. A method according to claim 1 wherein adjustingsaid at least one intermediate coefficient comprises adjusting said atleast one intermediate coefficient responsive to coefficients in aneighborhood of said at least one intermediate coefficient.
 26. A methodaccording to claim 25 wherein said neighborhood comprises a plurality ofintermediate coefficients, including said at least one intermediatecoefficient, for which any two intermediate coefficients have samevalues for all the same coordinates except one.
 27. A method accordingto claim 1 wherein adjusting said at least one intermediate coefficientcomprises adjusting said at least one intermediate coefficientresponsive to a ratio between a partial coefficient and a function ofother partial coefficients.
 28. A method according to claim 1 whereinadjusting said at least one intermediate coefficient comprises adjustingsaid at least one partial coefficient responsive to a trend in changesof values of partial coefficients as a function of changes in at leastone coordinate of said intermediate space.
 29. A method according toclaim 1 wherein adjusting said at least one intermediate coefficientcomprises adjusting said at least one partial coefficient responsive toan expected trend in changes of values of partial coefficients as afunction of changes in at least one coordinate of said intermediatespace.
 30. A method according to claim 1 wherein adjusting said at leastone intermediate coefficient comprises adjusting said at least onepartial coefficient responsive to a predetermined template.
 31. A methodaccording to claim 30 wherein said template identifies an edge in saidrecovered image and comprising smoothing said recovered image on eitherside of said identified edge.
 32. A method according to claim 1 whereinadjusting the value of said at least one intermediate coefficientcomprises adjusting the value of said at least one intermediatecoefficient responsive to data from different compressed data that isrelated to said compressed data.
 33. A method according to claim 32wherein said image in said image space comprises values for a firstcolor component of a color image in said image space and said differentcompressed data comprises data for generating a second color componentimage of said color image.
 34. A method according to claim 32 whereinsaid different compressed data identify an edge in said recovered image.35. A method according to claim 31 comprising smoothing said image oneither side of said edge.
 36. A method according to claim 1 comprisingtesting said set of adjusted intermediate coefficients for consistencywith a first set of intermediate coefficients defined in an intermediatespace different from said intermediate space in which said adjustedintermediate coefficients are defined.
 37. A method according to claim36 wherein testing said set of adjusted intermediate coefficientscomprises transforming said set of adjusted intermediate coefficientsinto a second set of intermediate coefficients in said differentintermediate space and comparing intermediate coefficients of said firstset with intermediate coefficients of said second set to determine ifsaid set of adjusted coefficients is consistent with said first set ofintermediate coefficients.
 38. A method according to claim 37 whereincomparing coefficients comprises determining how close the value of atleast one intermediate coefficient of said first set is to the value ofat least one intermediate coefficient of said second set.
 39. A methodaccording to claim 38 wherein determining how close the value of atleast one intermediate coefficient of said first set is to the value ofat least one intermediate coefficient of said second set comprises usinga metric.
 40. A method according to claim 36 wherein if said set ofadjusted intermediate coefficients is not consistent with said first setof intermediate coefficients, the value at least one adjustedintermediate coefficient of said set of adjusted intermediatecoefficients is readjusted to generate a readjusted set of intermediatecoefficients so that said set of readjusted intermediate coefficients isconsistent with said first set of intermediate coefficients.
 41. Amethod according to claim 1 comprising smoothing said recovered image.42. A method according to claim 1 wherein said image is an image definedin a three dimensional image space.
 43. A method according to claim 1wherein said image is an image defined in a two dimensional space.
 44. Amethod according to claim 1 wherein said compressed data is datagenerated using a unitary separable transform to transform an image insaid image space into a set of values in said transform space.
 45. Amethod according to claim 1 wherein said compressed data is generatedusing a JPEG compression method.
 46. A method according to claim 1wherein said compressed data is generated using an MPEG compressionmethod.
 47. A method according to claim 1 wherein said compressed datais generated using a PX64 compression method.
 48. A method according toclaim 1 wherein said compressed data is generated using an H261compression method.
 49. A method according to claim 1 wherein saidcompressed data is generated using an H263 compression method.
 50. Amethod according to claim 1 wherein said compressed data is generatedusing an H323 compression method.
 51. A method according to claim 1wherein said compressed data is generated using an HDTV compressionmethod.
 52. A method according to claim 34 comprising smoothing saidimage on either side of said edge.
 53. A method for recovering an imagedefined as a function of image coordinates in an image space fromcompressed data that is a function of transform coordinates in atransform space the method comprising: a) using said data to generate afirst set of recovered transform coefficients in said transform space;b) determining an adjustment range for at least one recovered transformcoefficient; c) replacing said at least one recovered transformcoefficient by a value in said adjustment range responsive to recoveredtransform coefficients in a neighborhood comprising a plurality of lessthan all of said recovered transform coefficients for which any tworecovered coefficients have same values for all the same coordinatesexcept one, to generate an adjusted set of recovered transformcoefficients; and d) recovering said image by transforming said adjustedset of recovered transform coefficients to said image space.